Skip to content

Fix CI; mask what leaves the service; an honest demo - #7

Open
arslanvuzmal wants to merge 4 commits into
mainfrom
demo/site-proof
Open

arslanvuzmal wants to merge 4 commits into
mainfrom
demo/site-proof

Conversation

@arslanvuzmal

Copy link
Copy Markdown
Owner

CI

  • Main has failed since d379eba with ModuleNotFoundError: No module named 'src'. pytest now adds the repository root to the path.
  • Locally: ruff clean, mypy clean, 54 passed (52 existing + 2 new).

Privacy fixes

  1. The reply step got the raw message. In both the Python engine and the TypeScript server, masking only covered the logs. The reply step now receives the masked text.
  2. The handoff summary sent raw messages to OpenAI. It's the one real model call (when OPENAI_API_KEY is set), and it sent the last ten messages unmasked. They're masked now, and a test intercepts the request to check.
  3. The ticket summary showed card numbers. The TypeScript summary is labelled "PII masked" but quoted them. Masked now.

Demo honesty

  • No invented history by default. At start-up the dashboard generated 60 random interactions, latencies, "GPT-4o" token counts and a 21.7% handoff rate, plus example tickets and sessions. It's now off unless SEED_SAMPLE_METRICS=true, and then labelled "Sample data".
  • The simulated reply step says it's simulated. It used to report "gpt-4o"/"gpt-4o-mini" and invented token counts, and told the customer an image had been analysed "using GPT-4o Vision". It now reports simulated and 0 tokens.
  • The "Refund > $500" preset now reaches the policy limit. It never did: the rule looked for the strings "500", "600" or "1000", and the preset says $650.00. The amount is now read from the message. The Python engine has the same string rule and is not changed here, because its tests rely on it.
  • Removed fixed claims: the "100% routed" and "Sub-second" tags, and "immutable" for an in-memory ledger.

README: what runs today, how to run the demo, and corrected feature rows (no LangChain, no Presidio, names are not masked).

Left for the owner: the Python engine also labels its simulated replies as gpt-4o, with fixed token counts and the same "GPT-4o Vision" text. Its tests assert on these, so it's left as it is.

🤖 Generated with Claude Code

arslanvuzmal and others added 4 commits October 20, 2026 08:22
CI on main has failed since d379eba: tests import the app as `src.…`, but
`pip install -e .` treats src/ as a source layout and never makes `src`
importable ("ModuleNotFoundError: No module named 'src'"). pytest now adds
the repository root. Locally: ruff clean, mypy clean, 52 passed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Privacy (each found while recording the demo):
- The reply step received the raw message: masking only covered the logs,
  in both the Python engine and the TypeScript server. It now gets the
  masked text (tests/test_model_input_masked.py).
- The handoff summary, the one real model call (OpenAI, when a key is set),
  sent the last ten messages unmasked. Masked now
  (tests/test_summary_masked.py). The TypeScript summary, labelled
  "PII masked", quoted card numbers; masked now.

Demo:
- The dashboard generated 60 random interactions, latencies, "GPT-4o"
  token counts and a handoff rate at start-up, plus example tickets and
  sessions, and regenerated them on reset. Now off unless
  SEED_SAMPLE_METRICS=true, and then labelled sample data.
- The simulated reply step reported "gpt-4o"/"gpt-4o-mini" and invented
  token counts, and told the customer it had analysed an image "using GPT-4o
  Vision". It now reports "simulated", 0 tokens, and says it is simulated.
- The "Refund > $500" preset never reached the policy: the rule looked for
  the strings "500", "600" or "1000" and the preset says $650.00. The amount
  is now read from the message.
- Removed fixed "100% routed" and "Sub-second" tags and "immutable" (the
  ledger is in memory).

README: what runs today (simulated reply step, masking scope, mock
helpdesk), how to run the demo, and corrected feature rows (no LangChain or
Presidio; names are not masked).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The docker-build job could not run: tags built from github.repository
contain capitals. Never seen before because lint-test failed first.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…raries)

Trivy's 4 HIGH findings (msgpack, urllib3, setuptools) are pip's own
vendored copies (pip/_vendor/vendor.txt), in both the venv and the system
interpreter. The service never installs at run time. Local Trivy run with
the CI settings: exit 0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant